Does Mentioning ‘‘some People’’ and ‘‘other People’’ in an Opinion Question Improve Measurement Quality?

Researchers often measure attitudes and beliefs using ‘‘some/ other’’ questions (‘‘Some people think that . . . but other people think that . . .’’) instead of asking simpler ‘‘direct’’ questions. This article reports studies testing the hypothesis that the some/other question form yields improved response validity. Eight experiments embedded in national surveys provided no evidence in support of this hypothesis. Instead, validity was found to be greatest when employing the direct question format presenting response options in an order that did not violate conversational conventions. Furthermore, some/other questions take longer to ask and answer and, because they involve more words, require greater cognitive effort from respondents. Therefore, the some/ other format seems best avoided; direct questions that avoid unconventional response option ordering are preferable to maximize data quality. Researchers often measure attitudes and beliefs by stating both sides of an issue before asking for the respondent s opinion. This type of question, which we call a ‘‘some/other’’ question, first says what ‘‘some’’ people believe, then says what DAVID YEAGER is an Assistant Professor in the Department of Psychology, University of Texas at Austin, Austin, TX, USA. At the time this work was conducted, he was a Ph.D. candidate at Stanford University, Stanford, CA, USA. JON KROSNICK is Frederic O. Glover Professor in Humanities and Social Sciences, Professor of Communication, Political Science, and Psychology (by courtesy) at Stanford University, and is University Fellow at Resources for the Future. The authors thank Edward Haertel, Gary Langer, Neil Malhotra, and the Political Psychology Research Group at Stanford University for their advice; the General Social Survey, Lightspeed Research, and Luth Research for including the experiments described here in their data collections; and Kinesis Survey Technologies for the use of their Internet questionnaire administration platform. Some data were collected by Time-Sharing Experiments for the Social Sciences, funded by the National Science Foundation [0818839 to Jeremy Freese and Penny Visser, Principal Investigators]. Other data were collected via the Face-to-Face Recruited Internet Survey Platform (the FFRISP), funded by the National Science Foundation [0619956 to J. A. K., Principal Investigator]. *Address correspondence to David Yeager, Department of Psychology, University of Texas at Austin, 1 University Station A8000, Austin, TX, 78712; e-mail: [email protected]. doi: 10.1093/poq/nfr066 Advance Access publication February 3, 2012 The Author 2012. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please e-mail: [email protected] ‘‘other’’ people believe, and then asks what the respondent believes. For example: ‘‘Some people say the American role in Vietnam was a noble cause. Other people say it was wrong and immoral. Do you think it was a noble cause, or wrong and immoral?’’ (CBS News 2000) This format has been used often in major surveys, including every American National Election Study (Krosnick, Lupia, and ANES 1948–2008) survey and every General Social Survey (GSS; Davis, Smith, and Marsden 1972–2008). Such questions have been included in many visible news media surveys as well. A search for questions containing both ‘‘some people’’ and ‘‘other people’’ in the survey archives maintained by the Inter-University Consortium for Political and Social Research (ICPSR) indicated that since 1950, 352 some/other questions have been asked in hundreds of national surveys. Of course, all some/other questions could be asked more succinctly in what could be called the ‘‘direct form,’’ such as ‘‘Do you think the American role in Vietnam was a noble cause, or wrong and immoral?’’ Some researchers have argued that the some/other form increases the accuracy of self-reports (Harter 1982; Schuman and Presser 1981). According to this view, respondents who hold socially undesirable opinions or who are simply in the minority on an issue may feel uncomfortable honestly reporting their attitudes (see Noelle-Neumann 1993). These researchers proposed that the some/ other format might increase validity because it communicates to respondents that about half of the people in the population hold one view while the other half hold the opposite view. By implying that unpopular views are more normative, these researchers argued, the some/other form might legitimize the unpopular choice and thereby help respondents who hold these unpopular opinions to feel more comfortable answering honestly. Hence, measurements might be more valid. As appealing as the some/other format might seem from this point of view, it might also cause unintended distortions of opinion reports. First, some/ other questions involve about twice as many words as direct questions, and longer questions can increase respondent burden and questionnaire administration time. On practical grounds, therefore, a direct question might seem preferable. 1. Some/other questions have been asked in a variety of different subforms. The ICPSR database reveals that the subform of some/other questions examined in the present study was among the most frequently used. This form ends with a request for a choice between two explicitly stated options, as in the examples above. 132 Yeager and Krosnick